From: Doug Lenat and Randy Davis

To: Denny Brown

cc:  Lee Hecht, Rick Hayes-Roth, and Dina Barr

Re:  Tut-1 the week of 21 June.

We had a number of comments and observations about the Tut-1
we just finished, with an eye toward incremental improvements for
next time.  Dina, Rick, and Lee are receiving copies since each had asked us
for comments and we responded by saying we were drafing a note.  Here
it is.

To provide the appropriate calibration, we give priority numbers for
each item, ranging from 1 (top) to 10 (bottom).  The 1's are crucial
to the success of future tutorials; the 10's are icing on the cake.

Effort is an independent dimension; we occassionaly calibrate it
explicitly, sometimes it is obvious. Some lower priority things are
very easy and so should probably be done even tho they're of lower
priority.



Perhaps the chief focus effort in getting ready for the next TUT-I
should be on the Lab.


Item: The script handed out to the students should be exactly reproducible,
      and reproducible in a reasonable period of time.
Priority: 1.

This has two important criteria: there must be no bugs (unimplemented
features, rules that cause infinite loops, etc.), and the program
must run fast enough so that they can expect to get through the
script in the allotted time.  Each of these is important.  

Imagine the image we give of our company if our toy expert system
doesn't work.  Would you buy any piece of software from a company that
can't even produce a trivial symbolic algebra system?  How can we claim
to sell genuine Roll Royces when we can't even build a toy car?

If the program is slow or the system crowded, etc., the script handed out
could be a pared down version of the original.  But it must be possible to
complete it in the allotted time.

If we are still using a high-load-average machine next time, we might
split the first lab session into halves, on Tue and Wed afternoon,
with half the class coming in each day, and getting the other
afternoon "off", rather than all of them getting Wed. afternoon off.

It is, we feel, important enough to ensure that the lab works, that
we should hold a meeting on the Monday a week before the tutorial
begins.  It should be attended by Denny (and all the relevant Minima
programmer(s)), Doug, and Rick.  At the meeting, we should run
significant parts of the script to see that it works exactly as
shown, and to calibrate the elapsed real time.  We should then
project as best we can the expected load average on the machine the
students will use, to be sure they can still get it done.

If lab is not ready, or is too slow, etc., then at that meeting we
should jointly arrive at a detailed plan for having some acceptable
recourse: changing the script, changing machines, going back to the
original MacLisp MINIMA, eliminating the lab sessions entirely,
hand-simulating them, etc.

The preferred way of ameliorating the lab problems is to improve the
program.  This means:
(i) speeding it up by at least a factor of 8; that would make it
comparable to the program we ran last TUT-I. An additional factor of
2-4 is only slightly less necessary.
(ii) adding the various features that were spec'ed out in the
original script:
	How, Why, Better-rule, Common subexpression,
	Subproblem-of, Best/Depth/Breadth-first, Tutoring,
	Caching subproblems, Organizing the rules into a hierarchy,
	Transforming subproblem-reoccurennce into a linear
	equation so that integration by parts can work usually, etc.

We believe that the program could best be sped up by recoding it in
Interlisp directly, eliminating the MRS mid-layer. The precise types
of pattern-matching and indexing needed for the script can be done
much more quickly than the version we had last week.  Caching
intermediate (sub-)problem results will also help, as will compiling,
of course.  MRS is an interesting research vehicle, but using it for
MINIMA is like using a MAC truck to run down a mouse: it's more than
we need and it's so much more that it actually makes things more
difficult to hit that mouse.


Item: The lab should be more interesting and informative to watch.  
Priority: 3

If it is indeed recoded in Interlisp directly, then it should take
only a few days of additional effort to get it to work on the Dolphins,
with separate windows dynamically displaying
	The body of rules (highlighting the one being used)
	The body of assertional facts, known subproblems, etc.
	The tree of subproblems (highlight the current one)

If the consensus is that not enough programmer time is available,
Doug will be happy to spend the time necessary.

We would need to have a total of about 20 Dolphin-hours set aside for
TUT-I that week, which should not be very painful as we can schedule
those hours months in advance.

Item:  Schedule a "Lee and Rick show" presentation as a regular part of
       the week.
Priority: 4

Given the "time off" periods (Tue and Wed afternoons, Tue for half
the class and Wed for the other half), we propose to use them to
provide an optional technical TEK pitch: the sort of talks that Lee
and Rick delivered on Friday: what exactly does our company sell and
do, who comprises it, what kinds of relationships do we want to set
up, etc.  Some students strongly want this, and some strongly don't,
so using the otherwise-time-off period for it seems about right.


Item:   Reduce the variance in the background and intended goals
        of the audience.  
Priority: 2

Some of these people are extremely well prepared in AI, Lisp, expert
systems, etc.; at the other extreme, many of them come from classical
DP backgrounds, with little motivation, litle willingness to open
their minds and challenge their paradigms. Some of the students will
be expected to become KEs; some are managers; some are scouts for
their managers and must gather ammunition with which to pitch a case
to enter ESs and/or buy Lisp machines.  For the scouts the focus is:
what's avaliable, what does it cost, who can use it, how much
manpower does it take, what are proven cases of cost-effectiveness,
etc.  To meet this problem, we propose that various flavors of TUT-I
be advertised and marketed, with specific audiences in mind:
	The manager or management scout
	The uninformed programmer
	The already-budding KE
Note that this is largely a marketing solution.  We will change our
offering to suit the various groups, but the important part is
simply recognizing the current problem, identifying relevant subgroups
(perhaps not exactly those we suggest above) and then making sure we
market appropriately.

This is extremely important for the success of future Tutorials. It
is hard to describe how wearing it is, for both us and the students,
to be teaching what we believe to be good material, well presented,
and discover that the students keep chafing at the bit, trying to get
us to talk about different topics.  We should be prepared to give
them what they want.  And the only way to do this is to make sure
they (and we) know what we're offering, why, and to whom.


Item:  We need hard information on issues and techniques for management
       of KE efforts.
Priority: 3

Given the strong interest in management and business details in
previous TUT-Is, and anticipating a stronger focus on those in the
Management-oriented future TUT-I's, we need hard data to present to
them.  Lee, Rick, Jerry, and others can help with this.

We need
	Case studies, with as much detail -- including dollar amounts
		and times -- as possible.
	Marketing studies
	Executive briefing materials (staffing a KE project, funding,...)
	Detailed specs on various Lisp machines, and on large
		machines which run various forms of Lisp.
	Samples of various Lisp environments.  This (and previous item)
		can probably be obtained from Fahlman and Steele at AAAI.
		They are giving a tutorial on Lisp environments just
		prior to AAAI; we should send someone to take it all down.
	Samples of running various extant ESs.  These samples can be
		hardcopy traces, living running programs, vidoetapes, etc.
	Sets of prepackaged alternatives for a company to adopt
		Dialect of Lisp
		Machine
		KE tools, languages, etc. for representationand control
		Mode of the project (e.g., collaboration, independent,...)
		Editor

Some of this was presented by Lee and Rick on Friday, and a less
ambitious version of this is suggested above (standardizing the talks
by Lee and Rick). But it is probably worth a serious effort for
future tutorials.


Item: Minor administrative problems and triumphs
Priority 6: 

This is a raft (sic) of observations on things that we noticed going
especially well or poorly at the level of logistics.

Having someone responsible for making the room shipshape every day
was a marked improvement over last time when it was nobody's job. The
food was good and always present, the coffee was always there. The
projectors were there and set up when we needed them. 

There was less communication (with us) than there could have been.
This ranged from the trivial to the truly major.  On the less
important end of the scale: We were not told, e.g., that the TUT was
being given in 525 Univ Ave this time; who the attendees (and company
affil and status) were; how and where to park (incl: getting enough
parking stickers for us as well as for the students), why we were
being videotaped, why 3 man-months were spent making TEX-ed versions
of our slides, why the MINIMA program was recoded into MRS-INTERLISP,
etc.  

On a more significant level, remember that we have four consultants
who are experienced teachers of AI:  Ed, Doug, Randy, and Mike.
It makes sense to make use of their time in designing future courses.

For the upper end of the scale, see below (a separate topic).

Clearly the company can't function if every decision has to be
discussed with us, but we're very far from that now, and feel that
we know too little of what's going on and why.  We don't necessarily
want to discuss everything, but we do want to know what's been decided
and why.  We might even have some advice and good suggestions.

More logistics:
The room is oddly shaped for our purpose, quite noisy (kitchen
noises), hot, and has no "second door" to allow us an unintrusive
entry/exit.  The slides were not ready until almost literally the
last minute, and as a result there was a collection of problems in
that area: some bugs on the slides, some lines that were supposed to
be hand-drawn not drawn, many slides poorly laid out compared to the
originals (TEX is to blame, but that's why we don't use it to prepare
slides).

Since we constantly revise and improve our slides, it's much easier
to have photocopies made of our version and have booklets printed of
photoreduced versions from those.  Getting the Tex right the first
time is an incredible amount of work; we are surprised it turned out
as well as it did.  Keeping it up to date is only a little less
tiresome, and, we feel, unnecessary.  It has been argued that uniformity
of style is important, and we believe that.  But we have to ask whether
the several man-months of effort it required and will continue to
require are worth the result.  There may be easier ways.

We'd be happy to supply the slides we're going to use, 10 days prior
to the start of each TUT-I.  We feel it's useful to have the
students' slide booklets conform exactly to the slides we actually
use on the screen. 

Badges: It's useful to have both person and Company name on them, including
	ours.  (We have to look different from the students somehow.
	Also, it may be obvious to us, but it isn't obvious to the students
	who belongs to Tek and who is a student.)  On the other hand,
	having "Director" and "Speaker" ribbons hang down seemed a
	bit pretentious.  It's obvious who's speaking, so having a
	ribbon is pointless.  And Denny is introduced as the director,
	so that one's not needed either.

Introductions: In keeping with being a serious, high-quality
organization, there should be serious, high-quality introductions.
Several students asked us during the week who exactly we were, where
we went to school, etc., so this information is more than a formality.

We have also found that the time right after lunch on day 1 is ideal
for having students introduce themselves, company, interest in ES, etc.
This takes a few minutes per student and is of great use both to
us in adjusting our talks and to the students in breaking the
ice, etc.   Doing this at the beginning of morning 1 (rather than
after lunch) puts the students on the spot
at a time when they're much more ready to listen than to talk.


Item: The banquet.
Priority 7: 

A formal dinner (sit-down) freezes you into conversation with the
same few people all evening.  A buffet-style meal permits more
circulation; can we try that instead?

There also seemed to be too many students and too few TEK people; a
ratio of about 1.5 / 1 (Student/TEK) seems about right.  (We seem to
be converging: in the very first banquet, TEK folk outnumbered
students.)

The private dining room is very nice and should be continued,
though we need not use such a fancy restaurant if we have more
people.


Item: Communication, high level issues.
Priority: 1

We are a fast-growing, constantly changing organization.  That's
exciting.  But it also puts a high premium on communication. There
really needs to be more attention paid to being sure that each of our
many hands knows what the other is doing.

We commented above on the damage to our credibility and image done by
an incomplete lab.  Friday afternoon an exchange occurred that was
only slighty less appalling:

  student: Where can we get EMycin?
  Randy:  Stanford will license an unmaintained version, but
	  Teknowledge has a version ...
  interruption by Doug: No we don't...
  Randy: oh yeah, that's right we don't sell it any more...
  Denny: <explanation about how we used to market it but don't any more>
  Dina: Yes, but we still might be convinced to sell it to
	  some "qualified" people under certain circumstances...

Every one of these comments was probably correct, but imagine the effect.
Any potential customer in the room must surely have decided to look
elsewhere, for a company that knew what it was doing.

We absolutely must get some communication going and a firm policy
established about what we're doing.  We need a "company line"; even
if it changes every three months; before the tutorial starts we have
to have an established set of answers.

We need more communication between:

marketing and the education division: 
Is what we offer in the tutorial what we're telling people we're
offering?  Should they arrive expecting a set of ideas or a set of
prepackaged software tools they will be taught to use during the
week?  Is TUT-I to interest clients, to raise our RollsRoyce image,...?

education and tutorial "faculty": 
Why are we videotaping? Is the quality of the tapes good enough for
the intended use?  We think not.  Randy got a flier recently that says
he'll be at the British Expert Systems Conference on Expert Systems
on videotape.  Doug is going to take 1-2 made during the previous
Tut-1 and show it/them, but didn't expect it to be "billed" in the
announcement of the offering!  Recall the bad lighting, diversionary 
questions from
the audience, amatuerish camera work (one camera constantly panning;
the problem is the setup, not the ability of the cameraman), and a
lecturer unprepared to be taped (if you look at the camera the live
audience is offended; if you look at the live audience the people
viewing the tape have the funny feeling you're talking to someone
else [which of course you are], and they're being excluded [which of
course they are]). Is this is quality of presentation Rolls Royce
would offer?  Any long-range use of videotapes as a packaged
product should be produced as a separate product, specially designed
and taped just for that mode of interaction and marketing.

marketing and faculty: 
What kinds of courses can we offer?  What courses do we have the
manpower and interest to staff?

On these and other issues, there must be more communication or we
will look like the 54 'Chebby of the expert systems world.


Well, there it all is.  Looks like there are no priority 9-10 items
(we never just make passing suggestions!).  

We're happy to comment further on any of this.
D&R